Added new OpenSSH break glass admin management operations guide #49804
Conversation
Signed-off-by: Alen Haric <[email protected]>
github-actions
bot
requested review from
mmcallister,
r0mant,
xinding33 and
zmb3
deusxanima
added
the
no-changelog
|
🤖 Vercel preview here: https://docs-5h5xxnof7-goteleport.vercel.app/docs |
docs/pages/admin-guides/management/operations/breakglass-access.mdx
Outdated
Show resolved
Hide resolved
| Run the following after logging into Teleport via `tsh login` as the impersonating user: | ||
|
|
||
| ```bash | ||
| tctl auth sign --ttl 24h --user=breakglass --out=/safe/path/breakglass --format openssh |
There was a problem hiding this comment.
This user certificate is (or, at least, I really hope is) issued by the user CA, which is the signer that signs all the user certs that are given out to users, but above we configured opensshd to trust the openssh CA, which signs certificates that are only ever in possession of the proxy - a regular user won't be able to obtain such a certificate.
There was a problem hiding this comment.
Interesting. From what I can see, it looks like the CA that's exported with the proxy via the https://proxy.example.com/webapi/auth/export?type=openssh endpoint, is of type "user" instead of OpenSSH according to the extension appended at the end:
e.g.:
cert-authority ssh-rsa <my_ca> clustername=alen.teleport.sh&type=user
I can confirm that passing type=user instead during the export produces a different CA, but if the former is not expected to work with user certs generated via tctl auth sign ... --format openssh, there may be another problem altogether as I can get it to work reliably right now (unless again I'm turned around on this and misunderstanding, which very likely could be the case)
|
Hi @deusxanima, would this PR address #3760? It's one of our oldest docs issues, so it'd be great to see it cleared! |
Partially, yes. Specifically it would cover the "If Teleport Auth cluster is offline" scenario in the original request. |
Signed-off-by: Alen Haric <[email protected]>
deusxanima
temporarily deployed
to
docs-amplify
GitHub Actions
Inactive
|
Amplify deployment status
|
Signed-off-by: Alen Haric <[email protected]>
deusxanima
temporarily deployed
to
docs-amplify
GitHub Actions
Inactive
docs/pages/admin-guides/management/operations/breakglass-access.mdx
Outdated
Show resolved
Hide resolved
docs/pages/admin-guides/management/operations/breakglass-access.mdx
Outdated
Show resolved
Hide resolved
docs/pages/admin-guides/management/operations/breakglass-access.mdx
Outdated
Show resolved
Hide resolved
docs/pages/admin-guides/management/operations/breakglass-access.mdx
Outdated
Show resolved
Hide resolved
Signed-off-by: Alen Haric <[email protected]>
deusxanima
temporarily deployed
to
docs-amplify
GitHub Actions
Inactive
| This guide will walk you through the steps required to configure emergency "break glass" access for critical agents and servers via OpenSSH using Teleport-issued certificates, in the following scenarios: | ||
|
|
||
| 1. The Teleport Agent on a server has crashed, gone offline, or become unusable. | ||
| 2. The Teleport control plane is down and cannot be used to access systems, and this procedure is necessary to fix it. |
There was a problem hiding this comment.
| 2. The Teleport control plane is down and cannot be used to access systems, and this procedure is necessary to fix it. | |
| 2. The Teleport control plane is down and cannot be used to access systems, and out-of-band access is necessary to fix it. |
Might just be me, but 2) makes it sound like you can do this procedure after the node goes offline, which you can't AFAIK. Maybe a slight change?
|
|
||
| ## How it works | ||
|
|
||
| Teleport's CA can issue short-lived, signed certificates that can be used to grant access to OpenSSH servers in emergency disaster recovery scenarios. By configuring the OpenSSH server on the Teleport Agent to trust Teleport’s CA, users with valid Teleport-issued certificates can authenticate to the server without requiring static SSH keys or passwords even if Teleport itself is down. |
There was a problem hiding this comment.
| Teleport's CA can issue short-lived, signed certificates that can be used to grant access to OpenSSH servers in emergency disaster recovery scenarios. By configuring the OpenSSH server on the Teleport Agent to trust Teleport’s CA, users with valid Teleport-issued certificates can authenticate to the server without requiring static SSH keys or passwords even if Teleport itself is down. | |
| Teleport's `openssh` CA can issue short-lived, signed certificates that can be used to grant access to OpenSSH servers in emergency disaster recovery scenarios. By configuring the OpenSSH server on the Teleport Agent to trust Teleport’s `openssh` CA, users with valid Teleport-issued certificates can authenticate to the server without requiring static SSH keys or passwords, even if Teleport itself is down. |
Clarify which CA
There was a problem hiding this comment.
❗
We're not configuring the openssh CA, we're specifically trusting the user CA in a way that hopefully doesn't break the security model of Teleport SSH. Users are never in possession of a usable certificate from the openssh CA.
| Now, `sshd` will trust users who present a Teleport-issued SSH certificate. | ||
|
|
||
| <Admonition type="important"> | ||
| Complete Steps 4 and 5 below to ensure only the `breakglass` user is allowed to log in with a valid Teleport certificate and is configured with proper access restrictions in order to prevent user switching once logged in. |
There was a problem hiding this comment.
| Complete Steps 4 and 5 below to ensure only the `breakglass` user is allowed to log in with a valid Teleport certificate and is configured with proper access restrictions in order to prevent user switching once logged in. | |
| Complete Steps 4 and 5 below to ensure only the `breakglass` user is allowed to log in with a valid Teleport certificate and is configured with proper access restrictions, in order to prevent user switching once logged in. |
| sudo adduser breakglass | ||
| ``` | ||
|
|
||
| Set a password for `breakglass` when prompted. |
There was a problem hiding this comment.
| Set a password for `breakglass` when prompted. | |
| Set a password for `breakglass` when prompted. We recommend the use of a long, random string as this password will never be used for access. |
| tctl auth sign --ttl 24h --user=breakglass --out=/safe/path/breakglass --format openssh | ||
| ``` | ||
|
|
||
| This command will create an ssh private key and a Teleport CA-signed certificate to be used by your OpenSSh client when authenticating with the OpenSSH server. |
There was a problem hiding this comment.
| This command will create an ssh private key and a Teleport CA-signed certificate to be used by your OpenSSh client when authenticating with the OpenSSH server. | |
| This command will create an ssh private key and a Teleport CA-signed certificate to be used by your OpenSSH client when authenticating with the OpenSSH server. |
| **Example:** Create a cron job to rotate certs every 20 hours and push them to a secure vault: | ||
|
|
||
| ```bash | ||
| 0 */20 * * * tctl auth sign --ttl 24h --user=breakglass --format openssh | vault kv put secret/breakglass breakglass=- >> /var/log/vault-cron.log 2>&1 |
There was a problem hiding this comment.
I don't really like us recommending vault here, but it's nice that we have a recommendation. I can't think how we'd improve it.
|
|
||
| Store the private key securely in a vault or other secure location, and only access if needed to implement break glass procedures. | ||
|
|
||
| <Admonition type="tip"> |
There was a problem hiding this comment.
I wonder whether this ought to be bigger/bolder? Old habits die hard, and anyone doing this as a one-off might just export one key/cert and think "right, we're done here" - then they're locked out when they come to need it.
There was a problem hiding this comment.
Maybe use <Admonition type="important"> instead, it's more striking on the page
Added guide with instructions on how to configure OpenSSH break glass access with Teleport CA signed certificates for out-of-band access if Teleport server resources or cluster servers become unavailable and/or unresponsive.